Proxies for speech generating devices

ABSTRACT

Speech generating devices, communication systems, and methods for communicating using the devices and systems are disclosed herein. In certain examples, the speech generating device includes: a first display device positioned in a first direction, the first display device configured to face a user of the speech generating device; an input device configured to generate a communication to be displayed on the first display device; and a second display device positioned in a second direction different from the first direction, wherein the second display device is configured to display a social cue proxy or an emotional proxy of the generated communication.

BACKGROUND

Many people with physical or cognitive challenges (e.g., severe speechimpairments) may use Augmentative and Alternative Communication (AAC)systems in order to express themselves. This may include individuals whosuffer from autism, amyotrophic lateral sclerosis (ALS), brain injuries,or strokes.

Certain AAC systems include speech generating devices. A speechgenerating device (SGD) acts as a proxy for conveying speech to thosenearby as a prosthetic for a loss of speech.

Certain SGDs may include an electronic interface with specializedsoftware configured to assist the user in creating a message that may betranslated into an electronic voice output. SGDs may produce theelectronic voice output by using digitized recordings of natural speechor through speech synthesis.

There are several challenges in communicating with a speech generatingdevice. One challenge is that the SGD may have a very slowconversational throughput rate, with users generally producing only 8-10words per minute. As such, there may be “dead time” during aconversation when the SGD user is inputting text into their device. Insome instances, this slow communication pace may prompt a conversationpartner to attempt to finish the SGD user's sentences for them, lookover their shoulder as they input characters or words into their device,or ask a second question before the user has finished preparing aresponse to the first question. Such interaction may be undesired by theSGD user.

Another challenge with a speech generating device is that the electronicvoice output of the SGD may not convey adequate emotion for the user. Inother words, while the electronic voice output may be more “natural” interms of sounding more like a human rather than prototypical “machine”speech, the output generally lacks emotive aspects of speech critical tocommunication.

Furthermore, an additional challenge with certain speech generatingdevices is being able to direct the inputted text to a distinctconversation partner. Certain SGDs only provide for broadcasting theinputted text at a predefined volume to nearby individuals around theuser of the SGD.

Various workarounds to these challenges include using text messaging oremail, or using something other than the SGD (e.g., eyebrow movements)to communicate the information or indicate the need for a more privateconversation. This can be problematic because a caregiver may not benearby, thus there is no ability to signal the caregiver. Texting oremail are non-ideal because they require too many additional actions bythe user, such as switching to a very different portion of the SGDsoftware interface, selecting the recipient, and so on. Further, not allSGDs support email or text.

SUMMARY

Communication systems, speech generating devices (SGDs), and methods ofusing communication systems and SGDs are described herein. In one ormore embodiments, a speech generating device is provided. The deviceincludes a first display device positioned in a first direction, thefirst display device configured to face a user of the speech generatingdevice. The device further includes an input device configured togenerate a communication to be displayed on the first display device.The device further includes a second display device positioned in asecond direction different from the first direction, wherein the seconddisplay device is configured to display a social cue proxy or anemotional proxy of the generated communication.

In another embodiment, a communication device is provided. Thecommunication device includes a speech generating device having a firstdisplay device configured to face a user of the speech generating deviceand an input device configured to generate a communication to bedisplayed on the first display device. The communication device furtherincludes a computing device separate from and wirelessly incommunication with the speech generating device, the computing devicecomprising a second display device configured to display a social cueproxy or an emotional proxy of the generated communication.

In another embodiment, a method is provided for communication with aspeech generating device. The method includes generating a communicationusing an input device of a speech generating device. The method furtherincludes displaying the communication on a first display device of thespeech generating device. The method further includes analyzing thecommunication using at least one processor and at least one memoryincluding computer program code for one or more programs of the speechgenerating device. The method further includes displaying a social cueproxy or an emotional proxy of the communication on a second displaydevice.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

DESCRIPTION OF THE DRAWING FIGURES

For a more complete understanding of the disclosure, reference is madeto the following detailed description and accompanying drawing figures,in which like reference numerals may be used to identify like elementsin the figures.

FIG. 1 depicts an example of a communication system having a speechgenerating device.

FIG. 2 depicts an example of a first display device and input device ofa speech generating device.

FIG. 3 depicts an example of communication requests and messages of afirst display device of a speech generating device.

FIG. 4 depicts an example of privacy and speech settings of a firstdisplay device of a speech generating device.

FIG. 5 depicts an example of privacy and history settings of a firstdisplay device of a speech generating device.

FIG. 6 depicts an example of a second display device of a speechgenerating device or communication system.

FIG. 7 depicts an additional example of a second display device of aspeech generating device or communication system.

FIG. 8 depicts an additional example of a second display device of aspeech generating device or communication system.

FIG. 9 is a flow diagram of a method of communicating using a speechgenerating device in accordance with a first example.

FIG. 10 is a flow diagram of a method of communicating using a speechgenerating device in accordance with a second example.

FIG. 11 is a flow diagram of a method of communicating using a speechgenerating device in accordance with a third example.

FIG. 12 is a block diagram of a computing environment in accordance withone example for implementation of the disclosed methods, one or moreSGDs, or communication systems.

While the disclosed devices, systems, and methods are representative ofembodiments in various forms, specific embodiments are illustrated inthe drawings (and are hereafter described), with the understanding thatthe disclosure is intended to be illustrative, and is not intended tolimit the claim scope to the specific embodiments described andillustrated herein.

DETAILED DESCRIPTION

Disclosed herein are speech generating devices (SGD), augmentative andalternative communication (AAC) systems, and methods for communicatingusing a SGD or AAC system. Such devices, systems, or methods haveseveral potential end-uses or applications, including improvingcommunication between a SGD user and a conversation partner, as well aspreserving the autonomy of the SGD user.

For example, the devices and systems described herein may address slowcommunication pace through use of social cue proxies. In certainexamples, a user of a SGD may provide a social cue proxy (1) indicatingthat the user is typing or composing a message, (2) indicating that theuser is speaking or about to speak, (3) indicating that the user isrequesting to speak, (4) requesting assistance, (5) identifying anemergency, (6) indicating that the user is calibrating the input deviceof the SGD (e.g., calibrating the eye gaze tracking device), (7)indicating that the SGD is idle.

Additionally, or alternatively, emotional proxies may be provided withthe SGD user's communication output (e.g., electronic voice output). Theuser may identify an emotion with the communication by selecting anemotion from a chart, emoji, image, graphic, or avatar on a display ofthe SGD. In other examples, emotion may be inferred by the SGD analyzingthe user's communication message. The selected or inferred emotion maybe translated into an emotional proxy and displayed on a display devicefor the conversation partner to see.

Further, in certain examples, the SGD user may direct the communicationmessage (and, e.g., the emotional or social cue proxy associated withthe communication message) toward a distinct conversation partner orgroup of conversation partners. The SGD user may also control the levelof sharing for the communication message for various groups or circlesof conversation partners as the SGD user is compositing the message.

In additional examples, one or more conversation partners may discreetlyassist the SGD user in developing the user's message. This assistancemay take place using a connected computing device that is part of thecommunication system, wherein the computing device is separate from theSGD device. The computing device (e.g., smartphone, tablet, laptopcomputer) may be connected through use of a software applicationinstalled on the computing device (e.g., computer program code), whereinthe application on the computing device is configured to communicatethrough a communication network (e.g., via a server) with the speechgenerating software application installed and operating on the SGD.

As used herein, the term “speech generating device” may refer to anelectronic communication device wherein text and/or multimedia areselected by a user to create a message or communication, and the messageis relayed to a conversation partner. The message may be relayedaurally, visually, or both. In some examples, the SGD is used by anindividual as a primary method of communication due to a severe speechimpairment. Nonetheless, use of the SGD is not limited to individualssuffering from speech impairments.

As used herein, the term “user” or “communicator” may refer to theindividual generating the communication message using the speechgenerating device.

As used herein, the term “conversation partner,” “receiver,” or“recipient” may refer to the individual or group of individualsreceiving a communication from the user or assisting the user ingenerating a message. A conversation partner may receive a communicationfrom a SGD user by listening to audio output from the SGD or a separatecomputing device, or by viewing a display screen of the SGD or a displayscreen of a separate computing device connected with the speechgenerating device within a communication system or communicationnetwork.

As used herein, the term “connected” may refer to a computing device ordisplay device that is in communication with the SGD via a communicationnetwork. The communication network may include wired networks, wirelessnetworks, or combinations thereof. The communication network may includea server configured to receive and transmit signals from the computingdevices and speech generating devices. A wireless network may be acellular telephone network, an 802.11, 802.16, 802.20, or WiMax network.Further, the network may be a public network, such as the Internet, aprivate network, such as an intranet, or combinations thereof, and mayutilize a variety of networking protocols now available or laterdeveloped including, but not limited to TCP/IP based networkingprotocols.

Speech Generating Devices and Communication Systems

FIG. 1 depicts a non-limiting a communication system 100 including aspeech generating device (SGD) 102. The SGD 102 includes a first displaydevice 104 configured to be positioned in front of or face a user 106 ofthe SGD 102. The communication system 100 further includes an inputdevice 108 configured to generate a communication on the first displaydevice 104. The communication system 100 may further include one or morespeakers to play the generated communication. In certain examples, theinput device 108 and speakers are part of the SGD 102.

The input device 108 may be an eye gaze tracking device; a pointingdevice such as a stylus, mouse, or joystick; a keyboard; an audio inputdevice; a video input device; a haptic input device; or a device forreceiving wired or wireless data transmissions. In certain examples, theinput device 108 is an eye gaze tracking device. The eye gaze trackingdevice may work in communication with the first display device 104,where text and/or multimedia (e.g., images, audio, video) displayed onthe first display device 104 are selected by the user 106 throughinteraction with the eye gaze tracking device. In other words, the user106 may select text and/or multimedia displayed on the first displaydevice 104 using the eye gaze tracking device to generate acommunication, which is also displayed on the first display device.Additionally, the eye gaze tracking device may be able to monitorwhether or not the eyes of the SGD user 106 are closed for at least acertain amount of time, which may provide an indication if the SGD user106 is sleeping.

The SGD 102 may further include one or more cameras configured to takepictures or video in the direction of the SGD user 106 or in theopposing direction away from the user 106. A camera positioned in thedirection of the user 106 may be configured to monitor the eyes of theuser 106 to determine if the user is awake or sleeping. The SGD 102 mayfurther include a processor and/or speech generating software (e.g.,computer program code) configured to analyze input from the input deviceto generate a message. Additional components of the SGD 102 aredescribed in detail below in the section “Exemplary ComputingEnvironment.”

The communication system 100 or SGD 102 may further include a seconddisplay device 110. In certain examples, the second display device 110is physically connected to or part of the SGD 102. In other examples,the second display device is part of the communication system 100 and isseparate from and wirelessly in communication with the SGD 102.

The second display device 110 may be positioned in a separate directionfrom the first display device 104. For example, the first display device104 may be positioned with a display screen projecting in a firstdirection 112 (e.g., toward the user 106), and the second display device110 may have a screen or display projecting in a second direction 114different from the first direction 112 (e.g., at an angle greater than0° from the first direction, at an angle between 90° and 270° from thefirst direction). In one example, the second display device 110 ispositioned on the opposite side of the SGD 102 such that the seconddisplay device 110 is positioned in a direction 114 at an angle 180°from the first direction 112.

The second display device and positioning of the second display deviceare advantageous in addressing certain challenges in communicationsbetween a user of a speech generating device and a conversation partner.For example, the second display device may identify a social cue oremotion of the SGD user to the conversation partner, provide an avenuefor discreet communication between the SGD user and conversationpartner, and/or allow the conversation partner to unobtrusively assistin constructing the SGD user's communication message. Further discussionregarding social cue and emotional proxies, avenues for discreetcommunications using a SGD, and avenues for interaction with the SGDuser during generation of the communication message are provided ingreater detail below.

FIGS. 2-5 depict different environments of a first display device of aspeech generating device. In particular, FIG. 2 depicts an environment200 having a first display device 204 and input device 208 of a speechgenerating device 202. In this non-limiting example, the first displaydevice 204 includes a display screen 206 depicting a QWERTY keyboard andshortcut keys. The keyboard keys 210 and/or shortcut keys 212 may beprogrammable based on various factors such as the user's definedpreferences, text recognition, or text prediction based on the contextof the word or sentence being formed. For example, text prediction maybe based on one or more factors: (1) the physical location of the SGD(as identified by a global positioning system (GPS) or any otherlocation identifying technology) and any surrounding context that may beinferred from the identified location, (2) a SGD user's previous use ordefined style of speaking in general, or (3) a SGD user's previous useor defined style of speaking with the specific conversation partner orgroup/circle of conversation partners. Additionally, or alternatively,one or more shortcut keys 212 may provide suggested text input from oneor more conversation partners (discussed in greater detail below).

The SGD user may generate a message using the keyboard keys 210 and/orshortcut keys 212. The generated message may be publicly played throughone or more speakers that are part of the SGD 202 or are connected tothe SGD 202. The user may publicly play the message by selection the“play” key 224. In other examples, the user may select the “private” key226 to discreetly send the message to one or more computing device ofconversation partners over a communication network (e.g., via a server).Alternatively, the private message may be sent via a text, email, ornotification to a computing device (e.g., second display device). Incertain examples, the SGD may be programmed to send a private message toa certain conversation partner based on the time of day that the messageis generated. In other examples, selection of the “private” key 226 mayopen a new window for the SGD user to select which conversation partneror social circle of conversation partners receives the message. In yetother examples, selection of the “private” key 226 may open a new windowfor the SGD user to select a social cue proxy to be displayed on thecomputing device or second display device identifying that theconversation partner should come to the SGD user to read the generatedmessage on the display screen 206. In yet other examples, selection ofthe “private” key 226 may play the generated message at a reduced volume(that is at a volume level that is less than the standard/default volumelevel for the SGD). These examples are advantageous as it allows the SGDuser to provide methods of public and discreet/private communicationavenues.

In certain examples, one or more of the shortcut keys 212 may beprogrammable for accessing and selecting a social cue or emotional proxyfor the user's generated communication. For example, a SGD user couldselect an “emotions” shortcut key 216 depicted in FIG. 2, wherein a newscreen would be displayed with a number of depicted emotions for theuser to choose from and attach with the generated message.

In additional examples, one or more of the shortcut keys 212 may beprogrammable to allow a SGD user to generate asynchronous or “offline”messages to share with one or more conversation partners when theconversation partner connects with the SGD user (e.g., when a computingdevice operated by the conversation partner connects with the SGD). Forexample, a SGD user could select the “share” shortcut key 218 on thedisplay screen 206 to open a selection of options for sharing a messagewith a potential conversation partner at a later date. Furtherdiscussion for generating an asynchronous message is provided below withFIG. 3.

In other examples, one or more of the shortcut keys 212 may beprogrammable to allow a SGD user to access a selection of options forhow messages are provided to conversation partners or which conversationpartners receive a generated message. For example, a SGD user couldselect a “settings” shortcut key 220 on the display screen 206 to open aselection of options for directing communications to conversationpartners, or for defining what conversation partners may see on theirdisplay device (e.g., the second display device). Further discussion fordirected or discreet communication is provided below with FIG. 4.

In certain examples, one or more of the shortcut keys 212 may beprogrammable to allow a SGD user to access a selection of options forwhat conversation partners may review regarding past messages generatedby a SGD user (e.g., whether a conversation partner may be able toreview all previous messages generated by the SGD user, the last or mostrecent message generated by the SGD user, or no historical messagesgenerated by the SGD user). For example, a SGD user could select the“settings” shortcut key 220 to open a selection of options forrestricting or granting access to conversation partners for previousgenerated messages. Further discussion for access to previouslygenerated messages is provided below with FIG. 5.

In certain examples, a conversation partner may discreetly assist theSGD user in composing the message in real-time. Through a connectedcomputing device within a communication system or network, theconversation partner may be able to view the SGD user's generatedmessage as it is being composed. In some examples, this may allow theconversation partner to view the message as it is being composedcharacter-by-character, word-by-word, or sentence-by-sentence. Theconversation partner may then be able to provide the user with suggestedhints for the next word, phrase, or sentence. Additionally, oralternatively, the conversation partner may be able to provide suggestedrevisions or corrections to incorrectly typed words, phrases, orsentences.

This process may be completed discreetly using the connected computingdevice, wherein the conversation partner enters the word, phrase, orsentence on the computing device and transmits the suggestion to theconnected speech generating device via a communication network (e.g.,via a server). The processor and compatible software (e.g., computerprogram code) of the SGD analyzes the suggestion (or suggestions)provided by the conversation partner (or multiple conversationpartners), and determines whether to replace one or more of thepredictive text shortcut keys 212 with the conversation partnersuggested text.

For example, as depicted in FIG. 2, the SGD user has begun composing amessage, “I am h . . . ” A conversation partner may provide one or moresuggestions for the next word. As depicted in this example, the top leftshortcut key 214 includes the word “hungry,” which was suggested by aconversation partner on a separate computing device, sent to the SGD,processed and inserted in the top left slot. Additional suggested textcould be inserted into one or more of the remaining five shortcut keyslots.

Suggested text (for predictions or corrections) may be regulated by theprocessor or software of the SGD. In certain examples, the SGD may beconfigured to regulate the number of suggestions possible. For example,a limited number of suggestions could be included (e.g., only one, two,or three suggestions from conversation partners maximum). In otherwords, the SGD may be configured (e.g., through its processor andsoftware application) to analyze and determine which suggested text toinclude or discard if more suggestions are provided than availableslots. For example, a suggestion from a conversation partner classifiedas a family member may be weighted more heavily than someone who is awork colleague. In other examples, the physical location of the SGD usermay dictate which conversation partner's suggestion is provided moreweight (e.g., a conversation in a hospital setting may weight asuggestion from a medical professional over family or friends).

The SGD may also be configured to grant or restrict certain conversationpartners from providing suggestions. For example, conversation partnersclassified as family or friends may be granted permission to providesuggested text additions or corrections, while work colleagues may not.Again, the physical location of the SGD user may dictate whether aconversation partner is permitted to provide suggestions. For example, awork colleague or medical professional may be granted permission to makesuggestions when the SGD user is at the office or at the hospital,respectively.

The SGD may also be configured to determine which shortcut key(s) toreplace with the suggested word(s), phrase(s), or sentence(s) (e.g., thefirst shortcut key in the top left of the display screen 206, the lastshortcut key in the top right of the display screen 206, or an internalshortcut key between the top left and the top right).

In certain examples, the suggested word, phrase, or sentence by theconversation partner may be highlighted or distinguished in some mannerfrom the remaining predictive text shortcut keys 212 to identify thatthe conversation partner provided suggestion (e.g., font, color, orshortcut input key size). In certain examples, the suggested text ishighlighted for a select number of conversation partners (e.g., familyand/or friends only). Again, the physical location of the SGD user maydictate whether a suggestion is highlighted in one of the shortcut keys212. For example, a suggestion from a work colleague or medicalprofessional may be highlighted when the SGD user is at the office or atthe hospital, respectively, as their suggestions may carry more weightin those environments.

In alternative examples, there is no distinguishing difference providedbetween a predictive text shortcut key and a suggested text shortcut key(e.g., the predictive text shortcut key and suggested text shortcut keyhave the same font, color, or shortcut input key size). This isadvantageous as these suggestions appear merged into the standardpredictive text selections on the SGD, thus allowing the user topreserve autonomy in choosing what to type, while at the same timeproviding contextually-appropriate suggested text options with thealgorithmically-based predictive text options.

In other words, the ability for a conversation partner to providesuggested text on the display screen 206 of the SGD 202 is advantageousas it provides a subtle or discreet interaction between the SGD user andconversation partner. The SGD user is not interrupted by conversationpartners guessing at what the user is typing. Additionally, the SGD userhas control over the degree to which they utilize the suggested textfrom their conversation partners. Further, this interaction between SGDuser and conversation partner has the benefit of further engagingconversation partners by providing them a method to directly interactwith the generated communication as it unfolds in real-time rather thansimply waiting for a block of communication to be completed beforehearing it spoken by the system's generated speech. Also, in examples ofproviding suggested corrections of typographical errors in the SGDuser's generated message, this may allow the SGD user to moreefficiently make changes to the generated message before sending themessage or playing it through the speakers of the device.

As depicted in FIG. 2, the input device 208 includes an eye gazetracking device 222 positioned below the first display device 204. Otherinput devices are also possible (e.g., track pads, keyboards), and theposition of the input device is configurable (e.g., positioned above thedisplay device, positioned to the left or right side of the displaydevice, or integrated with the display device itself). The input device208 may be physically connected to the first display device 204, e.g.,via a universal serial bus (USB) connection, IEEE 1394 (FireWire)connection, Ethernet connection, DisplayPort, mini DisplayPort, oranother physical connection technology, whether developed, indevelopment, or future developed.

In some examples, the input device 208 is not physically connected tothe first display device 204 of the speech generating device 202.Instead, the input device 208 may be remotely or wirelessly incommunication with or connected to the speech generating device 202. Thewireless connection may comply with a standard such as Bluetooth, IEEE802.11 (wireless LAN), ultra-wide band (UWB) radio link, or infrareddata association (IrDA) link.

To the extent the input device 208 includes an eye gaze tracking device222, an eye controller with one or more light sources and sensingelements may be provided relative to the display device to identify andcapture a user's selections. The first display device 204 may displayvisual objects that the user may select using the associated eye gazetracking device 222. A processor and eye gaze tracking softwareassociated with the speech generating device analyze data from the eyegaze tracking device 222 and select an object displayed on the displayscreen 206 of the display device 204. The tracking software may includean algorithm in conjunction with one or more selection methods to selectan object on the display screen 206 of the speech generating device 202by taking some action with one or both of the user's eyes.

Selection methods that may be activated using the eye gaze trackingdevice 222 to interact with the display screen 206 of the display device204 may include eye gaze tracking software that analyzes eye blinking,dwelling, switching, or a combination thereof (e.g., blink/dwell,blink/switch, dwell/switch). Using a blink selection method, a selectionwill be performed when the user gazes at an object on the display screen206 and then blinks for a specific length of time. The system may alsointerpret a “blink” as a set duration of time during which an associatedcamera cannot see the user's eye. The dwells selection method may beimplemented when the user's gaze is stopped on an object on the displayscreen 206 for a minimum defined length of time. The blink/dwellselection combines the blink and dwell selection so that the object onthe display screen 206 of the speech generating device 202 is selectedeither when the user's gaze is focused on the object for a specifiedlength of time or if before that length of time elapses, the user blinksan eye. In the switch selection method, an object is selected when theuser gazes on the object for a particular length of time and then closesan external switch. The blink/switch selection combines the blink andswitch selection so that the object on the display screen 206 isselected when the user's gaze blinks on the object and the user thencloses an external switch. Any number of commercial examples of eye gazetracking devices are applicable. One example of an eye gaze trackingdevice is a Tobii EyeX sensor (available from Tobii AB, Danderyd,Sweden).

FIG. 3 depicts an environment 300 including the display screen 206 ofthe first display device 204 of a speech generating device 202. In thisnon-limiting example, the display screen 206 depicts options for the SGDuser to generate and share messages 302 with a conversation partner orgroup of conversation partners. The display screen 206 of the firstdisplay device 204 may be accessed by a SGD user selecting a shortcutkey on the display device (e.g., a “share” shortcut key 218 depicted inFIG. 2). In such examples, after a user has generated a message 302, theuser may select an input key (e.g., a “share message” key 304 or a “saveas default” key 306) on the display screen 206 to share the generatedmessage with a selected conversation partner, a selected group ofconversation partners, or all conversation partners. In certainexamples, the message is shared on a second display device physicallyconnected to the SGD such that a conversation partner within viewingrange of the SGD (e.g., within the same room as the SGD user) may readthe generated message displayed on the second display device.

In other examples, the user's generated message may be shared on one ormore remotely or wirelessly connected display devices of selectedconversation partners. In some examples, the generated message is sharedwith a conversation partner when the SGD user makes a request to connectwith the conversation partner using the first display device of the SGD.Alternatively, a generated message may be shared with one or moreconversation partners currently connected with the SGD user.

In certain examples, the generated message is shared with a conversationpartner when the conversation partner connects with the SGD user ormakes a request to connect with the SGD user using a second displaydevice. In situations where an initial connection request has been madeby a potential conversation partner, the SGD user may approve or rejectthe request. For example, FIG. 3 depicts a pending request from apotential conversation partner “Jamie Doe” to connect with the SGD user.Upon accepting the request, the SGD user may define a relationshipcategory or social circle in which to place the conversation partner.This may include selecting a relationship category or social circle suchas “Family,” “Friends,” “Work,” “Medical,” “Other,” or any additionalcategory the SGD user may create with their SGD. Following thisrelationship designation, the user's generated message may be sent tothe second display device. In some examples, the SGD user may generatedifferent messages for different conversation partners or differentsocial circles. In such cases, a tailored message may be sent to theconversation partner upon connecting with the SGD user based on thedefined relationship between the two individuals (e.g., one message maybe sent to a work colleague conversation partner while a separatemessage may be sent to a medical personnel conversation partner).

The prepared or asynchronous message generated by the SGD user may beadvantageous for several reasons. For one, the asynchronous messageallows a SGD user to prepare communication content before a synchronousinteraction occurs. This may allow a conversation partner to read thegenerated message while the SGD user is constructing additional speechfor the current synchronous conversation, or the asynchronous messagemay offer content to fill a conversational gap that may occur due to thelow throughput of communication.

In certain examples, the generated asynchronous message may include: (1)communication preferences of the SGD user, (2) a pre-composed block oftext to initiate a conversation on a particular subject, (3) multimedia,or (4) activity data of the SGD user.

Regarding “communication preference” messages, this is a medium for theSGD user to express their preferred interactions and etiquette whencommunicating. For example, this could include messages such as, (1)“Please ask only yes or no questions;” (2) “Please do not look over myshoulder as I compose a message;” (3) “For private conversations, pleaseread over my shoulder so I do not have to display my thoughts for all tohear;” (4) “Please do not finish my thoughts for me,” or (5) “Pleaseassist me in constructing my message through your mobile phoneapplication.”

These preferences are particularly useful as a form of conversationpartner education, simplifying the process of instructing conversationpartners in the specific communication strategies and preferences of anygiven SGD user. In certain examples, the communication preferencesmessages may be displayed on the second display device the first time aconversation partner connects with the SGD user. In other examples, thecommunication preferences are displayed or provided on the seconddisplay device each time a conversation partner connects with the SGDuser, or each time a conversation partner connects with the SGD userafter a period of time has elapsed (as a reminder of the SGD user'scommunication preferences). In certain examples, different communicationpreference messages may be provided to different conversation partnersbased on the defined relationship between the SGD user and conversationpartner. For example, a SGD user may have certain communicationpreferences for work colleagues that differs from the communicationpreferences for medical personnel. Additionally, the SGD user may havecertain communication preferences for requesting assistance in makingcorrections or suggestions to their message as the user composes themessage in real-time (e.g., through a mobile phone software applicationthat is in communication with the SGD).

A second type of asynchronous message is a “pre-composed block” message.These are general communications that the SGD user wishes to share witha conversation partner, but would like to prepare ahead of a synchronousconversation. These messages are advantageous as they allow the SGD userto compose longer or more complex thoughts than may be possible toconstruct during synchronous communication. These pre-composed blockmessages may be displayed on the second display device for viewing by aconversation partner. In some examples, in the case of a remotely orwirelessly connected computing device, the pre-composed block messagemay be displayed on the display device of the separate computing devicewhen the conversation partner connects with the SGD user. Thisconnection may occur when the conversation partner opens an applicationon the computing device in preparation for having a conversation withthe SGD user. In certain examples, different pre-composed block messagesmay be provided to different conversation partners based on the definedrelationship between the SGD user and conversation partner.

A third type of asynchronous message is a “multimedia” message.Multimedia messages allow a SGD user to take pictures using their SGDdevice (e.g., through eye gaze selection of a camera-icon keyboard keyon the display screen). Multimedia messages also allow for a SGD user toselect any form of multimedia content (e.g., audio or video content).The multimedia content (e.g., picture) may be displayed on a seconddisplay device of the SGD or sent to a selected conversation partneroperating a separate computing device having a display device. This isadvantageous as the multimedia message provides the SGD user with a richchannel for sharing their experiences with others. This may increaseconversation throughput by reducing the need to type descriptions ofvisual scenes. In other words, “a picture is worth a thousand words.” Incertain examples, different multimedia may be provided to differentconversation partners based on the defined relationship between the SGDuser and conversation partner.

A fourth type of asynchronous message is an “activity data” message.Activity data may include, for example, a user's recent web browsinghistory, television- or movie-watching history, book reading orlistening history, music listening history, article reading history, orapplication usage history on their SGD or connected device. The activitydata may be collected and stored for a defined period of time (e.g., thelast hour, 2 hours, 4 hours, 8 hours, 12 hours, 24 hours, 1 week). Insome examples, the defined period of time may be the time interval sincethe SGD user and conversation partner last connected. The activity datafrom the defined period of time may be displayed on a second displaydevice of the SGD or sent to a selected conversation partner operating aseparate computing device having a display device. This is advantageousas the activity data message provides the SGD user with a channel forsharing their recent experiences with others. For example, theconversation partner may see that the SGD user watched a movie on theirdevice after they last connected. The conversation partner may be ableto skip the question “What have you been up to?” and dive right intoquestions about the movie. This may increase conversation throughput byreducing the need to have an introductory-type conversation. In certainexamples, different activity data messages may be provided to differentconversation partners based on the defined relationship between the SGDuser and conversation partner.

FIG. 4 depicts an environment 400 including a display screen 206 of afirst display device 204 of a speech generating device 202. In thisnon-limiting example, the display screen 206 depicts “levels of sharing”for the SGD user to define what a conversation partner or group ofconversation partners may be able to view as the SGD user generates amessage. The display screen 206 of the first display device 204 may beaccessed by a SGD user selecting a shortcut key on the display device(e.g., a “settings” shortcut key 220 depicted in FIG. 2).

This “levels of sharing” feature is advantageous as it allows a SGD userto decide what granularity of communication data the conversationpartner is able to view on a connected computing device having a displayscreen. This provides a SGD user with the ability to retain control orautonomy of how the user communicates with others.

Additionally, this feature provides a balance between two competingissues. First, generation of a message using the SGD may be relativelyslow and tedious. Therefore, requiring the SGD user to explicitly setprivacy permissions for every connected conversation partner may makethe system too tedious to use. Second, a SGD user may want to sharedifferent amounts of information with different types of conversationpartners (e.g., share more information with family and less with generalacquaintances).

Therefore, SGD users may classify conversation partners into socialcircles and set privacy permissions for entire circles rather thanindividual partners. As previously noted, a conversation partner may beinitially classified into a specific circle following the initialrequest and acceptance to communicate through the communication system.Examples of social circles include “Family,” “Friends,” “Work,”“Medical,” and “Other.” Additional circles may be programmable by theSGD user using the installed communication software on the SGD.

This classification of conversation partners into circles may functionboth as a user management feature and a privacy feature. In certainexamples, a potential conversation partner has to request to connectwith the SGD user through an application on their computing device andthen the SGD user has to approve the request and place the conversationpartner into a social circle. Only at this stage will the conversationpartner be able to see any conversations generated by the SGD user onthe connected speech generating device. In other examples, the SGD usermay make the initial request to connect with a conversation partner andestablish the social circle for the potential conversation partner.

As depicted in FIG. 4, a SGD user may use an interface to set thepermissions for an entire circle of conversation partners. This allowsthe SGD user to limit the amount of information presented in thereal-time view of synchronous messages on the computing device of thecircles of conversation partners. For example, the SGD user may allow aspecific group or circle of conversation partners to view updates for agenerated communication character-by-character, word-by-word,sentence-by-sentence, block-by-block (e.g., fully composed thoughts), orto only show status information (e.g., social cue information) and notshow text at all. In some examples, default settings may be provided foreach circle in order to reduce the amount of effort required from a SGDuser. For example, as depicted in FIG. 4, conversation partnersclassified as “Family” may view a SGD user's message as it is generatedcharacter-by-character, while “Friends” and “Medical” may receiveupdates word-by-word, “Work” may receive updates block-by-block, and“Other” may receive audio only.

FIG. 5 depicts an environment 500 including a display screen 206 of afirst display device 204 of a speech generating device 202. In thisnon-limiting example, the display screen 206 depicts “levels of sharing”for historical or previously generated messages. In this example, theSGD user may define what previous message or messages a conversationpartner or group of conversation partners may be able to view (e.g.,“All” of the SGD user's generated messages, only the “Last” or mostrecent message generated, or “None” of the user's generated messages).The display screen 206 of the first display device 204 may be accessedby a SGD user selecting a shortcut key on the display device (e.g., a“settings” shortcut key 220 depicted in FIG. 2).

This “levels of sharing” feature for a SGD user's history isadvantageous as it allows a SGD user to decide what level of context toprovide to a conversation partner or a group of conversation partners.This provides a SGD user with the ability to retain control or autonomyof how the user communicates with others. This feature also provides theability to update a more intimate conversation partner (e.g., family orfriend) on what the SGD user has been talking about recently, withouthaving to regenerate the message.

For example, as depicted in FIG. 5, conversation partners designated as“Family” may be able to view a SGD user's entire history of generatedmessages, allowing family members to catch up on any conversations thatthe user recently generated. Additionally, conversations partnersdesignated as “Friends” may be able to view a SGD user's last or mostrecently generated message, allowing friends to view what the SGD userwas recently talking about, possibly allowing the friend to develop aconversation with the SGD user about the same topic with some backgroundunderstanding in place.

FIGS. 6-8 depict various non-limiting examples of a second displaydevice of a communication system. In certain examples, the seconddisplay device is physically connected to and part of the SGD. In otherexamples, the second display device is separate from and remotely orwirelessly in communication with the SGD.

For example, FIG. 6 depicts an environment 600 of a second displaydevice 604 positioned on a surface of a SGD 602. The second displaydevice 604 is positioned such that the second display device 604 facesin a separate direction from the direction of the first display deviceof the SGD 602. In this example, the second display device 604 ispositioned along the x-z plane and the display is being directed alongthe y-axis extending from the page. The first display device may be onthe opposite surface of the SGD 602, wherein the first display device isalso positioned along the x-z plane and the display from the firstdisplay device is also directed along the y-axis, but extending into thepage.

The second display device 604 includes an array of lighting elements606. The elements 606 may be light emitting diodes (LEDs) (e.g., organicLEDs), or another emission technology, whether developed, indevelopment, or future developed, may be used. Each element of thelighting elements 306 may include an array of pixels (including, e.g., aplurality of subpixels) to display a variety of colors.

In this example, the array of lights 606 forms a circular shape on thesurface of the SGD 602. The array of lights 606 may be a strip of LEDlights. Other formations are also possible (see FIG. 4, for example). Inthe depicted example in FIG. 6, the plurality of lighting elements arephysically connected to the SGD via a connector 608 (e.g., cable). Thecable 608 may be externally connected on or above the surface of the SGD602 or internally connected beneath the surface of the SGD 602. Thecable 608 may be connected to the SGD 602 via a universal serial bus(USB) connection, IEEE 1394 (FireWire) connection, Ethernet connection,DisplayPort, mini DisplayPort, or another physical connectiontechnology, whether developed, in development, or future developed.

In alternative examples, the second display device 604 is not physicallyconnected to the SGD 602. Instead, the second display device 604 may beremotely or wirelessly in communication with or connected to the speechgenerating device 602. The wireless connection may comply with astandard such as Bluetooth, IEEE 802.11 (wireless LAN), ultra-wide band(UWB) radio link, or infrared data association (IrDA) link.

The second display device 604 may operate in conjunction with the firstdisplay device of the SGD. As the user is preparing a communication, orafter the user has completed and “spoken” or played the communicationthrough the speakers of the SGD 602, the second display device 604 mayidentify and display a proxy for a social cue or emotion of the SGD userregarding the communication.

The social cue or emotional proxy of the SGD user may be provided bydifferent lighting configurations or colors, for example. As depicted inFIG. 6, three lighting elements 610 are illuminated and are rotatedaround the circle, which may provide a social cue proxy to aconversation partner that the user is typing a message. Further examplesof social cue and emotional proxies are discussed in greater detailbelow.

As noted, the positioning of the second display device 604 may allow theconversation partner to view the user of the SGD 602 face-to-face like aconventional conversation, instead of looking over the shoulder of theuser as they operate the SGD 602. Additionally, through the social cueor emotional proxy identified on the second display device 604, theconversation partner may be aware of the user's communication status oremotion.

FIG. 7 depicts an additional environment 700 of a second display device704 positioned on a surface of a SGD 702. The environment in FIG. 7 issimilar to FIG. 6, except for the design of the lighting elements 706being in a rectangular shape. Through this design, social cue proxiesand emotional proxies may be identified in separate or similararrangements from the proxies used in the circular design of FIG. 6. Asdepicted in FIG. 7, several lighting elements 710 are illuminated toportray a smile, which may provide an emotional proxy to a conversationpartner that the user is happy or the message is meant to be humorous.Further examples of social cue and emotional proxies are discussed ingreater detail below.

In this depicted example in FIG. 7, the plurality of lighting elements706 are physically connected to the SGD via a connector 708 (e.g.,cable). The cable 708 may be externally connected on or above thesurface of the SGD 702 or internally connected beneath the surface ofthe SGD 702. The cable 708 may be connected to the SGD 702 via auniversal serial bus (USB) connection, IEEE 1394 (FireWire) connection,Ethernet connection, DisplayPort, mini DisplayPort, or another physicalconnection technology, whether developed, in development, or futuredeveloped.

Like the example in FIG. 6, the second display device 704 in FIG. 7 ispositioned such that the second display device 704 faces in a separatedirection from the direction of the first display device of the SGD 702.In this example, the second display device 704 is positioned along thex-z plane and the display is being directed along the y-axis extendingfrom the page. The first display device may be on the opposite surfaceof the SGD 702, wherein the first display device is also positionedalong the x-z plane and the display from the first display device isalso directed along the y-axis, but extending into the page.

FIG. 8 depicts an environment 800 of a second display device 804 of acommunication system or SGD. In this example, the second display device804 may be part of the SGD and positioned on a surface of a SGD.Alternatively, the second display device 804 depicted in FIG. 8 may bepart of a device physically separate from the SGD, wherein the separatedevice is part of the communication system and is remotely or wirelesslyconnected with the SGD. The wireless connection may comply with astandard such as Bluetooth, IEEE 802.11 (wireless LAN), ultra-wide band(UWB) radio link, or infrared data association (IrDA) link.

For example, the second display device 804 may be part of a computingdevice separate from the SGD. The computing device may be a personalcomputer (PC), server computer, tablet and other handheld computingdevice, laptop or mobile computer, communications device such as amobile phone (e.g., smartphone), multiprocessor system,microprocessor-based systems, set top box, programmable consumerelectronic, network PC, minicomputer, mainframe computer, or audio orvideo media player. In certain examples, the computing device may be awearable electronic device, wherein the device may be worn on orattached to a person's body or clothing. The wearable device may beattached to a person's shirt or jacket; worn on a person's wrist, ankle,waist, or head; or worn over their eyes or ears. Such wearable devicesmay include a watch, heart-rate monitor, activity tracker, orhead-mounted display.

As depicted in FIG. 8, the second display device 804 includes a displayscreen 806. The display screen 806 may include a light emitting devicesuch as an electronic-paper display, a liquid crystal display (LCD), alight emitting diode (LED) (e.g., an organic light emitting diode(OLED)), or a standard graphics display. The LCD or LED may be disposedin, or configured as, a film. The configuration, construction,materials, and other aspects of the light emitting devices may vary.Non-LED technologies, such as finely tuned quantum dot-based emissionstructures, may also be used. Other thin form factor emissiontechnologies, whether developed, in development, or future developed,may be used.

The display screen 806 of the second display device 804 may include anarray of pixels (including a plurality of subpixels) to display thevarious colors of an image. In certain examples, the second displaydevice 804 may operate in conjunction with the first display device ofthe SGD to display a social cue or emotional proxy of the user. Forexample, as the user is preparing a communication, or after the user hascompleted and “spoken” or played the communication through the speakersof the SGD, the second display device 804 may identify and display asocial cue or emotion of the SGD user regarding the communication on thedisplay screen 806.

The social cue or emotional proxy may be provided by different lightingconfigurations, colors, emoji, images, graphics, or avatars on thedisplay screen. For example, the background 808 of the display screen806 may be changed to a specific color (e.g., red) to function as anemotional proxy (e.g., anger) of the SGD user's message. Additionally oralternatively, the display screen 806 may provide context such as text810 of the social cue, e.g., “Bob is typing” or “Bob is talking.” Insome examples, the social cue may be provided in a unique pattern oflights or colors, such as those described above with FIGS. 7 and 8. Forexample, the indication that the SGD user is typing may be depicted by aseries of three dots 812 moving horizontally across the display screen806. Additional examples of social cue and emotional proxies arediscussed below.

The second display device 804 may also display helpful information 814to the conversation partner to assist in communicating with the SGDuser. For example, a SGD user may generate one or more asynchronous or“offline” messages 814 to share on the display screen 806 of the seconddisplay device 804. In certain examples, the asynchronous message isshared on the physically connected second display device 804 of the SGD.In other examples, the asynchronous message is shared on the remotely orwirelessly connected second display device 804 when the device connectswith the SGD (e.g., through a compatible application installed on thecomputing device, such as when the conversation partner opens theapplication to connect with the SGD).

In certain examples, as previously discussed, the generated asynchronousmessage 814 to be displayed on the display screen 806 may include: (1)communication preferences of the SGD user, (2) a pre-composed block oftext to initiate a conversation on a particular subject, (3) multimedia,or (4) activity data. Regarding “communication preference” messages,this is a medium for the SGD user to express their preferredinteractions and etiquette when communicating. These are particularlyuseful as a form of conversation partner education, simplifying theprocess of instructing conversation partners in the specificcommunication strategies and preferences of any given SGD user.Regarding “pre-composed block” messages, these messages are advantageousas they allow the SGD user to compose longer or more complex thoughtsthan may be possible to construct during synchronous communication.Additionally, “multimedia” messages may allow a SGD user to takepictures or video using their SGD device, providing the SGD user with arich channel for sharing their experiences with others, and thereinincreasing conversation throughput by reducing the need to typedescriptions of visual scenes. Further, activity data messages may allowa SGD user to share their recent experiences with others.

In some examples, the display screen 806 may provide the communicatedmessage in real-time as the message is being prepared by the user orafter the message has been completed and spoken through the speakers ofthe SGD or separate computing device. Based on the defined relationshipor social circle between the SGD user and communication partner, thereal-time message may be provided on the display screencharacter-by-character, word-by-word, sentence-by-sentence,block-by-block (e.g., fully composed thoughts), or to only show statusinformation (e.g., social cue information) and not show text at all.This is advantageous as it allows certain conversation partners to morefully engage with the SGD user in a synchronous communication. Thisprovides conversation partners with an accurate mental model of theongoing communication and potentially allows the conversation partner tobetter understand what the SGD user is attempting to say. Additionally,this feature provides that communication partners do not need to readover the shoulder of the SGD user when unsure of what the SGD user isattempting to write.

As previously noted, in certain examples, the second display device 804may be configured to assist the SGD user in composing the synchronousmessage. When the second display device 804 or computing device isconnected to the SGD, a conversation partner may be able to view the SGDuser's generated message as it is being composed. In some examples, thismay allow the conversation partner to view the message as it is beingcomposed character-by-character, word-by-word, or sentence-by-sentence.The conversation partner may then be able to provide the user withsuggested hints for the next word, phrase, or sentence. In other words,the conversation partner may enter words, phrases, or sentences on thesecond display device 804 and send them to the SGD for consideration.

For example, as depicted in FIG. 8, the display screen 806 indicatesthat the SGD user has begun composing a message, “I am h . . . ” Aconversation partner may enter one or more suggestions for the nextword, (e.g., “hungry,” “here,” “hurt”) using the second display device804 and send those suggestions to the SGD. This process is advantageousas the conversation partner is providing “discreet” suggestions to theSGD user through the connected devices. This respects the autonomy ofthe SGD user while leveraging the contextual knowledge and sharedhistory of the conversation partner to potentially improvecommunication.

In other words, the ability for a conversation partner to providesuggested words, phrases, or sentences on the SGD is advantageous as itprovides a subtle or discreet interaction between the SGD user andconversation partner. The SGD user is not interrupted by conversationpartners guessing at what the user is typing. Additionally, the SGD userhas control over the degree to which they utilize the suggested textfrom their conversation partners. Further, this interaction between SGDuser and conversation partner has the benefit of further engagingconversation partners by providing them a method to directly interactwith the generated communication as it unfolds in real-time rather thansimply waiting for a block of communication to be completed beforehearing it spoken by the system's generated speech.

In examples where the second display device 804 is part of a computingdevice separate from the SGD, the second display device 804 or computingdevice may include its own processor and/or software (e.g., computerprogram code) configured to communicate with the SGD. In certainexamples, software may be installed on the computing device (e.g.,mobile phone) that is compatible with communicating with the softwareinstalled on the SGD. In some examples, the software application may bedeveloped using HTML5 and JavaScript in the Apache Cordova framework,allowing it to run on a Windows Phone, Android phone, and iOS phone.Communication between the SGD software and the software application onthe separate communication device may be facilitated through a real-timeNoSQL database system.

Additionally, in examples where the second display device 804 is part ofa computing device separate from the SGD, the second display device 804or computing device may include one or more speakers configured to playa message generated by the SGD user on the SGD. In other examples, thesecond display device 804 or computing device may be connected to or incommunication with one or more speakers of a separate audio-playingdevice (e.g., headphones, wired or wireless speakers such as a Bluetoothaudio speaker) to play the device. This is advantageous as audioplayback on the computing device (e.g., mobile phone) or a connectedaudio-playing device allows the SGD user to choose how or where theirgenerated message is played (e.g., the conversation partner may notnecessarily be in the same room). In some examples, the SGD user mayselect to play their message publicly on their own device, wherein themessage is audible to other people within the same room or a certaindistance speaker(s) of the SGD. In other examples, the SGD user mayselect to play their generated message to a connected communicationpartner or a group of communication partners (e.g., within the samesocial circle). This is advantageous as it allows the SGD user to decidewhether or not they wish to have a public conversation, a private/sideconversation within the same room, or a remote conversation with aconversation partner in a different location than the user. In otherwords, in conjunction with a connected communication partner's abilityto receive real-time views of the user's generated messages, the mobileaudio provides telephony-like long distance communication for a SGDuser.

Social Cue Proxies

In certain examples, the user of the speech generating device may selecta social cue proxy to display on the second display screen. In otherexamples, the displayed social cue proxy may be determined based on aSGD processor/software analysis. As discussed in detail below, thesocial cue proxy may be (1) indicating that the user is typing orcomposing a message, (2) indicating that the user is speaking or aboutto speak, (3) indicating that the user is requesting to speak, (4)requesting assistance, (5) identifying an emergency, (6) indicating thatthe user is calibrating the input device of the SGD (e.g., calibratingthe eye gaze tracking device), (7) indicating that the SGD is idle, or(8) indicating that the SGD user's eyes are closed (e.g., sleeping).

To begin, one social cue proxy is an indicator that the user of the SGDis composing a message. This social cue proxy is advantageous in that itallows the conversation partner to know that the SGD user is activelyengaged in the conversation, or is attempting to initiate a conversationwith the conversation partner. The conversation partner may have askedthe SGD user a question, and may not be able to tell from the user'sfacial expressions or body movements (or lack thereof) whether the useris attempting to respond to the question. The indication that the useris typing or replying to the question potentially avoids theconversation partner from re-asking the question, asking a new question,moving over to look over the shoulder of the user, or leaving the roomprematurely. Furthermore, to the extent the second display deviceprovides no proxy that the user is actively typing, the conversationpartner may understand that he or she needs to repeat the question orexamine if the user's speech generating device is functioning properly.

In some examples, the user may select the social cue proxy to bedisplayed by the second display device. Alternatively, the processor ofthe SGD may analyze that the user is composing a message and display thesocial cue proxy indicating the user is composing a message. Forexample, once the user begins to compose the message using the inputdevice, the indicator or social cue proxy may be displayed on the seconddisplay device. The indicator may be a unique color, flashing lights orseries of light movements (e.g., three lights rotating around thecircular array of lighting elements in FIG. 3), or a message on a screenof the second display device indicating, (e.g., “Bob is typing”). Theindicator may remain displayed on the second display device as long asthe user is typing. The processor or software of the SGD may have apredefined time-out period, wherein the indicator remains displayed solong as the user is actively searching for the next word, character, orimage within the predefined time period (e.g., a character, word, orimage in 1 minute, 30 seconds, 15 seconds, 10 seconds, or 5 seconds).

A second social cue proxy is an indicator that the user of the SGD isspeaking or about to speak (e.g., the user is about to play a messagethrough the speakers of the SGD). This social cue proxy is advantageousin that it alerts the conversation partner to pay attention to the userand listen to the message. In this example, the user may select thesocial cue proxy to be indicated on the second display device, or theprocessor may provide the indication that the user is speaking or aboutto speak (e.g., after the user submits the message to be played throughthe speakers of the SGD). In certain examples, once the user submits themessage, the indicator may be displayed on the second display device.The indicator may be a unique color, flashing lights or series of lightmovements, or a message on a screen of the second display deviceindicating, e.g., “Bob is about to speak” or “Bob is speaking.” In someexamples, the second display device includes a plurality of illuminatedlights (e.g., LEDs) in a single color, wherein the intensity of thelights goes up and down as the message is played (e.g., wherein theintensity of the lights flows with the words being spoken through thespeakers). In other examples, in order to get the attention of theconversation partner, the second display may provide an indication(e.g., one or more flashes of light) that the user is about to speak.The indicator or proxy may remain displayed on the second display deviceas long as the user is speaking.

A third social cue proxy is a request to speak. This social cue proxy isadvantageous in that it allows the conversation partner (or partners) toknow that the SGD user would like to add something to the conversation.Due to particularly slower speeds of communication using a SGD, the usermay feel left out of a conversation with more than one conversationpartner. The user may have something to add to the conversation, but hasdifficulty “jumping in.” Through a social cue proxy of requesting tospeak, the user is “raising their hand” to be called on, allowing theiropinion to be heard. Like the previous examples, the user may directlymake a request to speak by selecting certain text or an image on thefirst display screen of the SGD, therein creating an indication on thesecond display device. Alternatively, the software or processor of theSGD may determine that the user would like to speak with theconversation partner. In some examples, the user may compose a messagebefore, during, or after the social cue proxy for a request to speak hasbeen initiated. Therefore, in some examples, after being acknowledged bythe conversation partner(s), the user may play their composed messagethrough the speakers of the SGD without noticeable delay in the speed ofthe conversation. Like the other previously described social cueexamples, the indicator for the request to speak may be a unique color,flashing lights or series of light movements, or a message on a screenof the second display device indicating, e.g., “Bob would like tospeak.” The indicator may remain displayed on the second display deviceuntil cleared by the user or the conversation partner/attendant, oruntil a predefined time has expired.

A fourth social cue proxy is a request for help or a request to read adiscreet message from a conversation partner, e.g., an attendant such asa nurse or hospital administrator. This social cue proxy is advantageousin that it allows the conversation partner to know that the SGD userneeds help from or would privately like to speak with the conversationpartner or attendant without having to play a message through thespeakers of the SGD where the message may be heard by others in thearea. In this example, the user may directly request help or request forthe conversation partner to read a discreet message through anindication on the second display device or move into a separate room fora private conversation. Alternatively, the software or processor of theSGD may determine that the user requires assistance from theconversation partner. For example, the SGD user may compose a messagerequesting assistance or a discreet conversation. The software andprocessor of the SGD may analyze the message and understand that help ora discreet conversation is being requested. Alternatively, help may berequested due to a perceived error in the SGD. For example, the speechgenerating device may not be operating properly (e.g., the speechgenerating software program may be frozen or a pop-up window on adisplay screen of the first display device may be preventing the userfrom composing a message.) The social cue proxy or indicator may bedisplayed on the second display device acknowledging that a request forhelp or a request for a discreet communication has been made. Theindicator may be a unique color (e.g., blue), flashing lights or seriesof light movements, or a message on a screen of the second displaydevice indicating, e.g., “Bob would like your assistance” or “Bob wouldlike to speak with you privately.” The indicator may remain displayed onthe second display device until cleared by the user or the conversationpartner/attendant, or until a predefined time has expired.

A fifth social cue proxy is an indication that there is an emergencyrequiring immediate attention. This social cue proxy is advantageous inthat it allows the conversation partner to know that the SGD user needsimmediate help. Again, the user may select the emergency indication tobe displayed on the second display device, or the processor of the SGDmay determine that an emergency situation exists. For example, the SGDmay be part of a communication system that includes other devices, suchas a heart rate monitor or other health monitoring equipment. The heartrate monitor or other health monitoring equipment may detect an anomalyin the user's health status, and the SGD may receive the indication ofthe anomaly. The SGD may then alert a conversation partner to theemergency issue via the second display device. The indicator on thesecond display device may be a unique color (e.g., red), flashing lightsor series of light movements, or a message on a screen of the seconddisplay device indicating, e.g., “EMERGENCY.” The indicator may remaindisplayed on the second display device until cleared by the user or anattendant, until a predefined time has expired, or until the healthmonitoring equipment indicates a non-emergency reading.

A sixth social cue proxy is an indication that the user is attempting tocalibrate the input device of the SGD. For example, the user mayoccasionally need to calibrate an eye gaze tracking device or atouchscreen input, for example, to create or maintain accurate input.This social cue proxy is advantageous in that it allows the conversationpartner to know that the SGD is busy, and that it may be some timebefore the user is able to generate a message or respond to a questionfrom the conversation partner. In this example, the processor orsoftware of the SGD may determine that the device is undergoingcalibration testing. The indicator on the second display device may be aunique color indicating the input device of the SGD is being calibrated,flashing lights or series of light movements, or a message on a screenof the second display device indicating, e.g., “CALIBRATING DEVICE.”

A seventh social cue proxy is an indication that the SGD is idle. Thissocial cue proxy is advantageous in that it allows the conversationpartner to know that the SGD is functioning properly, and that the userof the SGD device is not providing any other social cue proxy (e.g., theuser is not typing a message or requesting assistance). In this example,the processor or software of the SGD may determine that the device isidle based on a lack of activity for a predefined time period (e.g., atleast 5 seconds, at least 10 seconds, at least 15 seconds, at least 30seconds, at least 1 minute). The indicator on the second display devicemay be a unique color indicating the SGD is idle, flashing lights orseries of light movements, or a message on a screen of the seconddisplay device indicating, e.g., “IDLE.”

An eighth social cue proxy is an indication that the SGD user's eyes areclosed (e.g., sleeping). This social cue proxy may be determined usingthe eye gaze tracking device or a camera attached to or part of the SGD,wherein the eye gaze tracking device or camera monitors eye movement ofthe user and may determine if the user's eye lids are closed for atleast a period of time. This is advantageous in that it allows theconversation partner to know that the SGD user is not paying attentionor has fallen asleep. In this example, the processor or software of theSGD may determine that the SGD user's eye lids are closed (e.g., and theuser is sleeping) based on monitoring of closed eyes for a predefinedtime period (e.g., at least 15 seconds, at least 30 seconds, at least 1minute, at least 5 minutes). The indicator on the second display devicemay be a unique color indicating the SGD user is sleeping, flashinglights or series of light movements, or a message on a screen of thesecond display device indicating, e.g., “SLEEPING.”

Additional social cue proxy examples are also possible, and may beprogrammed into the SGD software based on a user's particularcommunication preferences.

Emotional Proxies

In addition to, or in the alternative to social cue proxies, the user ofthe speech generating device may display a proxy for the user's emotionon the second display device of the SGD or communication system. Theemotional proxy may be selected by the user or inferred through ananalysis of the user's generated communication.

In certain examples, the user may select from a chart or table ofemotions displayed on the first display device when preparing acommunication message. The chart or table of emotions may be retrievedfrom a database or software program of the SGD. For simplicity and easeof selection, the chart or table may include a limited number ofemotions from which to choose. In some examples, the user may modify thechart of emotions within the database based on a preference foridentifying certain emotions.

In other examples, when preparing a communication message, the user mayselect an emoji displayed on the first display device as a proxy for theuser's emotions. For example, the user could select a smiley face emojito represent happiness, a sad face emoji with a tear drop to expressgrief, an angry red face emoji to represent anger, and so on. Again, forsimplicity and ease of selection, the list of emoji to choose from maybe a limited number of options to display and select from a singlescreen.

In yet other examples, the user of the SGD may select an image, graphic,or avatar displayed on the first display device as a proxy for theuser's emotions. For example, the user could select an image of someoneyelling to express anger, an image of someone laughing to expresshappiness, an image of a rainstorm to express sadness, an image of aquiet beach to express calmness, and so on. The image or simple graphicmay be selected from a limited number of options for simplicity and easeof selection.

Alternatively, instead of the user providing an emotional proxy, anemotion may be inferred from the user's prepared message. Software ofthe SGD device may analyze the user's message for key words or phrasesthat may be tied to certain emotions. For example, if the user's messagesays “I hate onions,” the software may identify the word “hate” toconnote anger. The SGD may then identify that an emotional proxy foranger could be displayed on the second display device.

Following input from the user or inference by the SGD, the emotionalproxy may be displayed on the second display device. The emotional proxymay be displayed or portrayed by a unique color, image, graphic, emoji,avatar, or text description, for example.

In one example, the user's selected or inferred emotion is linked with aspecific color or color pattern. For example, the user may select theemotion from a chart, emoji, image, graphic, or avatar on the firstdisplay device, and the second display device display the emotionalproxy in the form of a unique color. Shades of red may represent passionor anger; shades of orange may represent warmth or joyfulness; shades ofyellow may represent energy or happiness; shades of green may representpeace or trust; shades of cyan may represent calmness or sadness; shadesof magenta may represent boredom or disgust. If the second displaydevice is an array of lighting elements (e.g., LEDs), the LEDs areilluminated with the identified color. If the second display deviceincludes a display screen, the display screen or portion thereof isilluminated with the identified color. In certain examples, theintensity of the color goes up and down as the user's message is played(e.g., wherein the intensity of the lights flows with the words beingspoken through the speakers). The identified color may remain displayedon the second display device until cleared by the user or until apredefined time has expired.

In additional examples, the user's selected or inferred emotion islinked with an emoji, image, graphic, or avatar. For example, after theuser has selected the emotion on the first display device or the emotionhas been inferred, the second display device may display the emotionalproxy in the form of a unique emoji, image, graphic, or avatar. If thesecond display device is an array of lighting elements (e.g., LEDs), theLEDs may be illuminated in the form of an emoji or graphic such as asmiley face or a wink. If the second display device includes a displayscreen, the display screen or portion thereof may display the emoji,image, graphic, or avatar. The identified emotional proxy may remaindisplayed on the second display device until cleared by the user oruntil a predefined time has expired.

Exemplary Methods

FIG. 9 depicts an exemplary method 900 for communicating using a speechgenerating device. At act S101, a communication is generated using aninput device of the SGD. The input device may be an eye gaze trackingdevice. In certain examples, the user of the SGD may select images,emoji, or graphics displayed on the first display device to identify asocial cue or emotion.

At act S103, the communication is displayed on the first display deviceof the SGD.

At act S105, the communication is analyzed by a processor of the SGD orcommunication system. The analysis may include inferring emotion fromtext generated by the user or from selected emotion imagery, emoji, orgraphics selected by the user.

At act S107, a social cue proxy or an emotional proxy of thecommunication is displayed on a second display device of thecommunication system. In some examples, the second display device ispart of and physically connected to the SGD. In other examples, thesecond display device is remote from and wirelessly in communicationwith the SGD.

In certain examples, an emotion is identified by the processor in theanalyzing act, and the identified emotion is displayed on the seconddisplay device as the emotional proxy. In other examples, a social cueis identified by the processor in the analyzing act, and the identifiedsocial cue is displayed on the second display device as the social cueproxy.

FIG. 10 depicts an exemplary method 1000 for communicating using aspeech generating device. At act S201, a SGD receives a request toconnect with a computing device of a potential conversation partner.This process may include installing a compatible software application onthe computing device (e.g., mobile phone, tablet, laptop computer) and,through the application, making the request to connect with the softwareapplication of the SGD over a wireless communication network (e.g., viaa server). Alternatively, in some examples, the SGD may make the requestto connect the device with the computing device of the potentialconversation partner.

At act S203, a connection is established between the speech generatingdevice and the computing device. In some examples, the SGD user approvesthe conversation partner's request to connect. In alternative examples,the conversation partner approves the user's request to connect.

At act S205, the SGD user designates a conversation group for thecomputing device (and the associated conversation partner of thecomputing device). The conversation group may be a relationship categoryor social circle in which to place the conversation partner. This mayinclude selecting a relationship category or social circle such as“Family,” “Friends,” “Work,” “Medical,” “Other,” or any additionalcategory the SGD user may create with their SGD. In some examples, theSGD user may also define a level of sharing for each conversation groupfor real-time updates of generated messages. In additional examples, theSGD user may define a level of sharing for each conversation group forhistoric or previous generated messages.

At act S207, the SGD user may generate a message with the SGD. Themessage or communication may be generated with an input device of theSGD. As noted above, the input device may be an eye gaze trackingdevice.

At act S209, the SGD user may transmit the generated message to aselected conversation group via a communication network, and anyconversation partners and computing devices designated as part of thegroup. In some examples, the transmission is to all of the deviceswithin the group that are currently connected over the communicationnetwork with the SGD. In certain examples, the computing devices notcurrently connected may receive a transmission when the device becomesconnected at a later time.

FIG. 11 depicts an exemplary method 1100 for communicating using aspeech generating device. At act S301, a SGD user composes or generatesan asynchronous or “offline” communication. The generated communicationmay be transmitted to and received by a server within the communicationnetwork. In certain examples, as previously discussed, the generatedasynchronous message may include (1) communication preferences of theSGD user, (2) a pre-composed block of text to initiate a conversation ona particular subject, (3) multimedia, or (4) activity data.

At act S303, the speech generating device (with SGD user) connects withat least one computing device (with associated conversation partner).The connection may be made established by a server over a wirelesscommunication network, wherein a software application of the SGD and acompatible software application of the computing device are connected.In some examples, the computing device is a mobile phone (e.g.smartphone), tablet, or laptop computer.

At act S305, at the time the connection is made or after the connectionis made, the communication is transmitted from the SGD or the server tothe computing device via communication network. In some examples, theSGD user initiates the transmission. In other examples, the serverinitiates the transmission when the server identifies the connection, orat a period of time after the identification of the connection.

Exemplary Computing Environment

With reference to FIG. 12, the speech generating device, separatecomputing device, server, and/or communication system, as describedabove, may be an exemplary computing environment 1200 or incorporatedwithin the exemplary computing environment. The computing environment1200 may correspond with one of a wide variety of computing devices,including, but not limited to, personal computers (PCs), servercomputers, tablet and other handheld computing devices, laptop or mobilecomputers, communications devices such as mobile phones, multiprocessorsystems, microprocessor-based systems, set top boxes, programmableconsumer electronics, network PCs, minicomputers, mainframe computers,or audio or video media players.

The computing environment 1200 has sufficient computational capabilityand system memory to enable basic computational operations. In thisexample, the computing environment 1200 includes one or more processingunit(s) 1210, which may be individually or collectively referred toherein as a processor. The computing environment 1200 may also includeone or more graphics processing units (GPUs) 1215. The processor 1210and/or the GPU 1215 may include integrated memory and/or be incommunication with system memory 1220. The processor 1210 and/or the GPU1215 may be a specialized microprocessor, such as a digital signalprocessor (DSP), a very long instruction word (VLIW) processor, or othermicrocontroller, or may be a general purpose central processing unit(CPU) having one or more processing cores. The processor 1210, the GPU1215, the system memory 1220, and/or any other components of thecomputing environment 1200 may be packaged or otherwise integrated as asystem on a chip (SoC), application-specific integrated circuit (ASIC),or other integrated circuit or system.

The computing environment 1200 may also include other components, suchas, for example, a communications interface 1230. One or more inputdevices 1240 (e.g., eye gaze tracking device; cameras; pointing devicessuch as a stylus, mouse, or joystick; keyboards; audio input devices;video input devices; haptic input devices; or devices for receivingwired or wireless data transmissions) may be provided. The input devices1240 may include one or more touch-sensitive surfaces, such as trackpads. Various output devices 1250, including touchscreen ortouch-sensitive display(s) 1255, may also be provided. The outputdevices 1250 may include a variety of different audio output devices,video output devices, and/or devices for transmitting wired or wirelessdata transmissions. The output devices 1250 may also include one or morespeakers for playing a generated message with the speech generatingdevice.

The computing environment 1200 may also include a variety of computerreadable media for storage of information such as computer-readable orcomputer-executable instructions, data structures, program modules, orother data. Computer readable media may be any available mediaaccessible via storage devices 1260 and includes both volatile andnonvolatile media, whether in removable storage 1270 and/ornon-removable storage 1280. Computer readable media may include computerstorage media and communication media. Computer storage media mayinclude both volatile and nonvolatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer readable instructions, data structures, program modules orother data. Computer storage media includes, but is not limited to, RAM,ROM, EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which may be used to store the desired informationand which may accessed by the processing units of the computingenvironment 1200.

In the above described examples, the communication network may includewired networks, wireless networks, or combinations thereof. Thecommunication network may include a server configured to receive andtransmit signals from the computing devices and speech generatingdevices. The wireless network may be a cellular telephone network, an802.11, 802.16, 802.20, or WiMax network. Further, the network may be apublic network, such as the Internet, a private network, such as anintranet, or combinations thereof, and may utilize a variety ofnetworking protocols now available or later developed including, but notlimited to TCP/IP based networking protocols.

While the computer-readable medium is described to be a single medium,the term “computer-readable medium” includes a single medium or multiplemedia, such as a centralized or distributed database, and/or associatedcaches and servers that store one or more sets of instructions. The term“computer-readable medium” shall also include any medium that is capableof storing, encoding or carrying a set of instructions for execution bya processor or that cause a computer system to perform any one or moreof the methods or operations disclosed herein.

In a particular non-limiting example, the computer-readable medium mayinclude a solid-state memory such as a memory card or other package thathouses one or more non-volatile read-only memories. Further, thecomputer-readable medium may be a random access memory or other volatilere-writable memory. Additionally, the computer-readable medium mayinclude a magneto-optical or optical medium, such as a disk or tapes orother storage device to capture carrier wave signals such as a signalcommunicated over a transmission medium. A digital file attachment to ane-mail or other self-contained information archive or set of archivesmay be considered a distribution medium that is a tangible storagemedium. Accordingly, the disclosure is considered to include any one ormore of a computer-readable medium or a distribution medium and otherequivalents and successor media, in which data or instructions may bestored.

In an alternative example, dedicated hardware implementations, such asapplication specific integrated circuits, programmable logic arrays andother hardware devices, may be constructed to implement one or more ofthe examples described herein. Applications that may include theapparatus and systems of various examples can broadly include a varietyof electronic and computer systems. One or more examples describedherein may implement functions using two or more specific interconnectedhardware modules or devices with related control and data signals thatcan be communicated between and through the modules, or as portions ofan application-specific integrated circuit. Accordingly, the presentsystem encompasses software, firmware, and hardware implementations.

In accordance with various examples of the present disclosure, themethods described herein may be implemented by software programsexecutable by a computer system. Further, in certain examples,implementations may include distributed processing, component/objectdistributed processing, and parallel processing. Alternatively, virtualcomputer system processing can be constructed to implement one or moreof the methods or functionality as described herein.

Although the present specification describes components and functionsthat may be implemented in particular examples with reference toparticular standards and protocols, the invention is not limited to suchstandards and protocols. For example, standards for Internet and otherpacket switched network transmission (e.g., TCP/IP, UDP/IP, HTML, HTTP,HTTPS) represent examples of the state of the art. Such standards areperiodically superseded by faster or more efficient equivalents havingessentially the same functions. Accordingly, replacement standards andprotocols having the same or similar functions as those disclosed hereinare considered equivalents thereof.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a standalone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatuses can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

As used in this application, the term “circuitry” or “circuit” refers toall of the following: (a) hardware-only circuit implementations (such asimplementations in only analog and/or digital circuitry) and (b) tocombinations of circuits and software (and/or firmware), such as (asapplicable): (i) to a combination of processor(s) or (ii) to portions ofprocessor(s)/software (including digital signal processor(s)), software,and memory(ies) that work together to cause an apparatus, such as amobile phone or server, to perform various functions) and (c) tocircuits, such as a microprocessor(s) or a portion of amicroprocessor(s), that require software or firmware for operation, evenif the software or firmware is not physically present.

This definition of “circuitry” applies to all uses of this term in thisapplication, including in any claims. As a further example, as used inthis application, the term “circuitry” would also cover animplementation of merely a processor (or multiple processors) or portionof a processor and its (or their) accompanying software and/or firmware.The term “circuitry” would also cover, for example and if applicable tothe particular claim element, a baseband integrated circuit orapplications processor integrated circuit for a mobile phone or asimilar integrated circuit in server, a cellular network device, orother network device.

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor receives instructions and data from a read only memory or arandom access memory or both. The essential elements of a computer are aprocessor for performing instructions and one or more memory devices forstoring instructions and data. Generally, a computer also includes, oris operatively coupled to receive data from or transfer data to, orboth, one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. However, a computer need nothave such devices. Moreover, a computer can be embedded in anotherdevice, e.g., a mobile telephone, a personal digital assistant (PDA), amobile audio player, a Global Positioning System (GPS) receiver, to namejust a few. Computer readable media suitable for storing computerprogram instructions and data include all forms of non-volatile memory,media and memory devices, including by way of example semiconductormemory devices, e.g., E PROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto opticaldisks; and CD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, examples of the subject matterdescribed in this specification can be implemented on a device having adisplay, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display)monitor, for displaying information to the user and a keyboard and apointing device, e.g., a mouse or a trackball, by which the user canprovide input to the computer. Other kinds of devices can be used toprovide for interaction with a user as well; for example, feedbackprovided to the user can be any form of sensory feedback, e.g., visualfeedback, auditory feedback, or tactile feedback; and input from theuser can be received in any form, including acoustic, speech, or tactileinput.

Examples of the subject matter described in this specification can beimplemented in a computing system that includes a back end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back end, middleware, or front end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While the present claim scope has been described with reference tospecific examples, which are intended to be illustrative only and not tobe limiting of the claim scope, it will be apparent to those of ordinaryskill in the art that changes, additions and/or deletions may be made tothe disclosed embodiments without departing from the spirit and scope ofthe claims.

The foregoing description is given for clearness of understanding only,and no unnecessary limitations should be understood therefrom, asmodifications within the scope of the claims may be apparent to thosehaving ordinary skill in the art.

Claim Support Section

In a first embodiment, a speech generating device comprises a firstdisplay device positioned in a first direction, the first display deviceconfigured to face a user of the speech generating device, an inputdevice configured to generate a communication to be displayed on thefirst display device, and a second display device positioned in a seconddirection different from the first direction, wherein the second displaydevice is configured to display a social cue proxy or an emotional proxyof the generated communication.

In a second embodiment, with reference to the first embodiment, theinput device is an eye gaze tracking device.

In a third embodiment, with reference to the first or second embodiment,the emotional proxy or the social cue proxy is displayed as a color oran arrangement of lights on a display screen of the computing device,the color or the arrangement of lights configured to identify an emotionor social cue.

In a fourth embodiment, with reference to any of embodiments 1-3, thesecond display device comprises an array of lighting elements.

In a fifth embodiment, with reference to any of embodiments 1-4, thesecond display device comprises an electronic-paper display, a liquidcrystal display, or a light emitting diode display.

In a sixth embodiment, with reference to any of embodiments 1-5, thesecond display device comprises a display screen configured to displaycommunication instructions for a conversation partner of the user of thespeech generating device.

In a seventh embodiment, with reference to any of embodiments 1-6, thespeech generating device further comprises at least one processor and atleast one memory coupled to the at least one processor, the at least onememory including computer program code for one or more programs; the atleast one memory and the computer program code configured to, with theat least one processor, cause the speech generating device to: analyzethe generated communication; and identify an emotion from the generatedcommunication, wherein the identified emotion is displayed on the seconddisplay device as the emotional proxy.

In an eighth embodiment, with reference to any of embodiments 1-7, thespeech generating device further comprises at least one processor and atleast one memory coupled to the at least one processor, the at least onememory including computer program code for one or more programs; the atleast one memory and the computer program code configured to, with theat least one processor, cause the speech generating device to: analyzethe generated communication; and identify a social cue from thegenerated communication, wherein the identified social cue is displayedon the second display device as the social cue proxy.

In a ninth embodiment, with reference to any of embodiments 1-8, thesocial cue proxy is (1) an indication that the user is typing orcomposing a message, (2) an indication that the user is speaking orabout to speak, (3) an indication that the user is requesting to speak,(4) a request for assistance, (5) an identification of an emergency, (6)an indication that the user is calibrating the input device of thespeech generating device, (7) an indication that the speech generatingdevice is idle, or (8) an indication that the user is sleeping.

In a tenth embodiment, with reference to any of embodiments 1-9, thefirst display device and the second display device are positioned withina single housing, and wherein the first direction is at a 180° anglefrom the second direction.

In an eleventh embodiment, a communication device comprises a speechgenerating device having a first display device configured to face auser of the speech generating device and an input device configured togenerate a communication to be displayed on the first display device;and a computing device separate from and wirelessly in communicationwith the speech generating device, the computing device comprising asecond display device configured to display a social cue proxy or anemotional proxy of the generated communication.

In a twelfth embodiment, with reference to the eleventh embodiment, thecomputing device is a personal computer, tablet computer, laptopcomputer, mobile phone, or wearable electronic device.

In a thirteenth embodiment, with reference to the eleventh or twelfthembodiment, the emotional proxy or the social cue proxy is displayed asa color or an arrangement of lights on a display screen of the computingdevice, the color or the arrangement of lights configured to identify anemotion or social cue.

In a fourteenth embodiment, with reference to any of embodiments 11-13,the communication device further comprises at least one processor and atleast one memory coupled to the at least one processor, the at least onememory including computer program code for one or more programs; the atleast one memory and the computer program code configured to, with theat least one processor, cause the communication system to: analyze thegenerated communication; and identify an emotion from the generatedcommunication, wherein the identified emotion is displayed on the seconddisplay device as the emotional proxy.

In a fifteenth embodiment, with reference to any of embodiments 11-14,the communication device further comprises at least one processor and atleast one memory coupled to the at least one processor, the at least onememory including computer program code for one or more programs; the atleast one memory and the computer program code configured to, with theat least one processor, cause the communication system to: analyze thegenerated communication; and identify a social cue from the generatedcommunication, wherein the identified social cue is displayed on thesecond display device as the social cue proxy.

In a sixteenth embodiment, a method of communicating comprisesgenerating a communication using an input device of a speech generatingdevice, displaying the communication on a first display device of thespeech generating device, analyzing the communication using at least oneprocessor and at least one memory including computer program code forone or more programs of the speech generating device, and displaying asocial cue proxy or an emotional proxy of the communication on a seconddisplay device.

In a seventeenth embodiment, with reference to the sixteenth embodiment,an emotion is identified in the analyzing, and wherein the identifiedemotion is displayed on the second display device as the emotionalproxy.

In an eighteenth embodiment, with reference to the sixteenth orseventeenth embodiment, a social cue is identified in the analyzing, andwherein the identified social cue is displayed on the second displaydevice as the social cue proxy.

In a nineteenth embodiment, with reference to any of embodiments 16-18,the speech generating device comprises the second display device.

In a twentieth embodiment, with reference to any of embodiments 16-19,the communication comprises an emotion or social cue input by the inputdevice.

1. A speech generating device comprising: a first display devicepositioned in a first direction, the first display device configured toface a user of the speech generating device; an input device configuredto generate a message to be displayed on the first display device; atleast one processor; at least one memory coupled to the at least oneprocessor, the at least one memory including computer program code forone or more programs, wherein the at least one memory and the computerprogram code are configured to, with the at least one processor, causethe speech generating device to: analyze the generated message, andidentify an emotion or a social cue from the generated message; and asecond display device positioned in a second direction different fromthe first direction, wherein the second display device is configured todisplay a social cue proxy or an emotional proxy of the generatedmessage representing the identified emotion or social cue, respectively.2. The speech generating device of claim 1, wherein the input device isan eye gaze tracking device.
 3. The speech generating device of claim 1,wherein the emotional proxy or the social cue proxy is displayed as acolor or an arrangement of lights on a display screen of the seconddisplay device, the color or the arrangement of lights configured torepresent the identified emotion or social cue.
 4. The speech generatingdevice of claim 1, wherein the second display device comprises an arrayof lighting elements.
 5. The speech generating device of claim 1,wherein the second display device comprises an electronic-paper display,a liquid crystal display, or a light emitting diode display.
 6. Thespeech generating device of claim 1, wherein the second display devicecomprises a display screen configured to display communicationinstructions for a conversation partner of the user of the speechgenerating device.
 7. The speech generating device of claim 1, whereinthe emotion is identified from the generated message, and wherein theidentified emotion is displayed on the second display device as theemotional proxy.
 8. The speech generating device of claim 1, wherein thesocial cue is identified from the generated message, and wherein theidentified social cue is displayed on the second display device as thesocial cue proxy.
 9. The speech generating device of claim 8, whereinthe social cue proxy is (1) an indication that the user is typing orcomposing the message, (2) an indication that the user is speaking orabout to speak, (3) an indication that the user is requesting to speak,(4) a request for assistance, or (5) an identification of an emergency.10. The speech generating device of claim 1, wherein the first displaydevice and the second display device are positioned within a singlehousing, and wherein the first direction is at a 180° angle from thesecond direction.
 11. A communication device comprising: a speechgenerating device comprising: a first display device configured to facea user of the speech generating device; an input device configured togenerate a message to be displayed on the first display device; at leastone processor; and at least one memory coupled to the at least oneprocessor, the at least one memory including computer program code forone or more programs; the at least one memory and the computer programcode configured to, with the at least one processor, cause the speechgenerating device to: analyze the generated message, and identify anemotion or a social cue from the generated message; and a computingdevice separate from and wirelessly in communication with the speechgenerating device, the computing device comprising a second displaydevice configured to display the social cue proxy or the emotional proxyof the generated message representing the identified emotion or socialcue, respectively.
 12. The communication device of claim 11, wherein thecomputing device is a personal computer, tablet computer, laptopcomputer, mobile phone, or wearable electronic device.
 13. Thecommunication device of claim 11, wherein the emotional proxy or thesocial cue proxy is displayed as a color or an arrangement of lights ona display screen of the second display device of the computing device,the color or the arrangement of lights configured to represent theidentified emotion or social cue.
 14. The communication device of claim11, wherein the emotion is identified from the generated message, andwherein the identified emotion is displayed on the second display deviceas the emotional proxy.
 15. The communication device of claim 11,wherein the social cue is identified from the generated message, andwherein the identified social cue is displayed on the second displaydevice as the social cue proxy.
 16. A method of communicatingcomprising: generating a message using an input device of a speechgenerating device; displaying the generated message on a first displaydevice of the speech generating device; analyzing the generated messageusing at least one processor and at least one memory including computerprogram code for one or more programs of the speech generating device;identifying an emotion or a social cue in the analyzing; and displayinga social cue proxy or an emotional proxy of the generated message on asecond display device representing the identified emotion or social cue,respectively.
 17. The method of claim 16, wherein the emotion isidentified in the analyzing, and wherein the identified emotion isdisplayed on the second display device as the emotional proxy.
 18. Themethod of claim 16, wherein the social cue is identified in theanalyzing, and wherein the identified social cue is displayed on thesecond display device as the social cue proxy.
 19. The method of claim16, wherein the speech generating device comprises the second displaydevice.
 20. The method of claim 16, wherein the generated messagecomprises the emotion or the social cue input by the input device.